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ABSTRACT 

In  2007,  the  U.S.  Department  of  Defense  (DoD)  released 
a  report  detailing  the  future  of  robotic  military  equipment 
and  how  to  proceed  in  development  and  procurement  of 
unmanned  systems  (Office  of  Secretary  of  Defense, 
2007).  This  document  recognizes  the  role  of  unmanned 
systems  in  the  areas  of  reconnaissance,  surveillance,  and 
target  identification.  This  role  for  unmanned  systems 
implies  a  need  for  improved  warfighter  support  and 
training.  The  Mixed  Initiative  Experimental  (MIX) 
Testbed  is  a  distributed  simulation  environment  which 
provides  a  means  for  this  type  of  training.  This  paper 
describes  the  design  and  capabilities  of  the  MIX  Testbed 
and  exemplar  scenarios  for  research  and  training  with 
unmanned  systems  of  varying  capabilities. 

1.  INTRODUCTION 

To  address  the  growing  use  of  unmanned  systems  in 
military  operations,  the  U.S.  Department  of  Defense 
(DoD)  released  a  report  detailing  the  future  of  robotic 
military  equipment  and  how  to  proceed  in  their 
development  and  procurement.  This  document.  The 
Unmanned  Systems  Roadmap,  incorporates  master  plans 
for  unmanned  air,  ground,  undersea,  and  surface  systems 
over  the  next  twenty  five  years  into  a  comprehensive 
roadmap  for  future  prioritization  of  development  and 
DoD  needs  (Office  of  Secretary  of  Defense,  2007).  This 
report  recognizes  the  increasing  role  of  unmanned 
systems  in  the  areas  of  reconnaissance,  surveillance,  and 
target  identification.  This  role  implies  the  need  for 
improved  warfighter  support  and  training  with  this  new 
technology  and  its  increasing  capabilities.  Moreover,  the 
roadmap  identifies  interoperability  between  unmanned 
systems  as  a  requirement  for  reducing  procurement  and 
training  costs. 


With  the  capabilities  of  unmanned  systems  growing  (e.g. 
autonomous  navigation),  researchers  need  to  investigate 
how  warfighter  performance  is  affected.  For  example, 
does  an  increase  in  the  level  of  automation  result  in  a 
decrease  in  situational  awareness?  There  has  been  much 
research  conducted  in  the  area  of  human  team 
performance  (Dixon  and  Wickens,  2003;  Driskell, 
Radtke,  and  Salas,  Sims,  and  Burke,  2005),  but  there  still 
remain  questions  for  optimizing  operational  systems  for 
“mixed-initiative”  teams.  Mixed-initiative  interactions 
refer  to  a  flexible  interaction  strategy  in  which  each  agent 
(human  or  computer)  contributes  what  is  best  suited  at  the 
most  appropriate  time  (Hearst,  Allen,  Guinn,  and  Horvitz, 
1999). 

The  MIX  Testbed  is  a  distributed  simulation  environment 
for  investigation  into  how  unmanned  systems  are  used 
and  how  automation  affects  performance.  The  MIX 
Testbed  includes  an  Operator  Control  Unit  (OCU), 
Unmanned  Vehicle  Simulator  (UVSIM),  and 
interoperability  with  existing  DoD  simulators  using  HLA 
and  the  Joint  Architecture  for  Unmanned  Systems 
(JAUS).  The  default  simulated  unmanned  system 
included  with  the  testbed  is  an  Unmanned  Ground  System 
(UGS).  This  platform  includes  a  camera  payload,  and 
supports  multiple  levels  of  automation.  The  OCU 
provides  a  standard  interface  for  users  to  control  any  type 
of  unmanned  system  supporting  JAUS  like  the  simulated 
UGS.  Users  can  send  mission  plans  or  teleoperate  the 
platform  with  a  joystick  while  being  provided  a  video 
feed  from  the  camera  payload;  tasks  typical  in 
reconnaissance  and  surveillance. 

To  directly  support  experimentation,  the  MIX  Testbed 
includes  extensive  data  and  event  logging  capabilities  in 
addition  to  neurophysiological  devices  for  measuring  a 
user’s  state.  Devices  that  measure  heart  rate, 
Electrocardiogram  (EKG),  Electroencephalogram  (EEG), 
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Galvanic  Skin  Response  (GSR),  and  gaze  location  can  be 
used  to  measure  workload  and  user  performance  in 
conjunction  with  data-driven  methods  within  the  test 
environment  (Nicholson,  Stanney,  Fiore,  Davis, 
Fidopiastis,  Finkelstein,  Arnold,  2006). 

The  work  presented  here  discusses  on-going  research  at 
the  University  of  Central  Florida’s  Institute  for 
Simulation  and  Training  targeted  at  investigating  mixed- 
initiative  teams  and  unmanned  systems  operator 
performance,  sponsored  by  the  U.S.  Army  Research  Lab 
(ARL)  and  U.S.  Army  Research  Development  and 
Engineering  Command  (RDECOM).  It  presents  the 
design,  implementation,  and  capabilities  of  the  MIX 
Testbed  and  how  it  is  used  for  research  in  Human-Robot 
Interactions  (HRI)  for  reconnaissance,  surveillance,  and 
target  identification. 

Section  2  of  this  paper  provides  the  design  and  goals  of 
the  MIX  testbed.  Section  3  describes  the  implementation 
of  the  main  components  which  include:  Operator  Control 
Unit  (OCU)  and  Unmanned  Vehicle  Simulator  (UVSIM). 
Section  4  describes  how  the  capabilities  of  the  MIX 
Testbed  are  used  in  example  scenarios  for  HRI  research. 

2.  DESIGN 

In  order  to  meet  the  goals  of  training  and  experimentation 
with  mixed-initiative  teams  the  MIX  Testbed  is  designed 
with  two  primary  requirements:  interoperability  with 
existing  distributed  simulations  and  measurement  of  user 
performance  and  state.  By  maximizing  interoperability 
with  existing  DoD  simulations,  the  MIX  Testbed  can  add 
functionality  that  is  missing  without  having  to  redevelop 
existing  capabilities.  Experimentation  is  supported  by 
including  data  logging  capabilities  into  MIX  components 
that  record  simulation  events,  user  interactions,  and 
neurophysiological  sensor  data. 

The  MIX  Testbed  includes  three  primary  components: 
UVSIM,  OCU,  and  Joint  Semi- Automated  Forces  (JSAF). 
UVSIM  simulates  different  types  of  unmanned  systems 
such  as  ground  or  air  vehicles.  The  OCU  allows  users  to 
interact  with  unmanned  systems  generated  by  UVSIM  for 
reconnaissance,  surveillance,  and  target  identification 
missions.  Finally,  JSAF  generates  simulated  entities  that 
simulated  unmanned  systems  and  other  compatible 
simulations  can  interact  with. 

Interoperability  within  the  MIX  Testbed  is  accomplished 
using  two  standards:  HLA  and  JAUS.  HLA  is  a  DoD 
standard  used  in  many  distributed  simulations  allowing 
MIX  components  to  work  with  existing  simulations  like 
those  included  with  the  Deployable  Virtual  Training 
Environment  (DVTE),  (Dahmann,  1997;  Bailey  and 
Armstrong,  2002).  JAUS  is  a  component-based  message 
passing  architecture  that  describes  services  that  unmanned 


systems  support,  and  is  recommended  by  the  Unmanned 
Systems  Roadmap.  Moreover,  JAUS  is  independent  of 
any  specific  hardware  or  technology,  allowing  operators 
to  learn  one  set  of  commands  that  can  be  used  on  any  type 
of  unmanned  system  (JAUS  Working  Group,  2008).  An 
additional  component  is  included  in  the  system  design  to 
bridge  HLA  and  JAUS  traffic,  allowing  HLA  Federates  to 
interact  with  components  using  JAUS  as  shown  in  Figure 
1. 


Figure  1  -  Distributed  simulation  design 


The  OCU  application  is  the  primary  interface  used  by 
participants  for  human  robot  interactions.  It  incorporates 
interfaces  to  the  neurophysiological  sensors  (e.g.  heart 
rate  monitor,  eye  tracker)  used  to  measure  a  users’  state. 
Sensor  data,  scenario  events,  and  user  interactions  are 
time  stamped  and  logged  for  each  experiment  by  the 
OCU. 

3.  IMPLEMENTATION 

3.1  Deployable  Virtual  Training  Environment  (DVTE) 

DVTE  is  a  training  and  testing  environment  that  provides 
distributed  simulation  capabilities.  Developed  for  the 
U.S.  Marine  Corps,  it  is  a  deployable  suite  of  simulators 
that  run  on  identical,  commercially  available  laptop 
computers.  One  component  of  DVTE  called  the  General 
Simulator  (GENSIM)  is  an  application  that  provides  a 
three-dimensional,  networked,  HLA  environment 
encapsulating  different  positions  within  Fire  Support 
(FiST)  and  Close  Air  Support  (CAS)  teams.  Team 
positions  that  can  be  trained  or  tested  include  mortar  and 
artillery  Forward  Observers  (FO),  Forward  Air 
Controllers  (FAC),  FiST  Leaders,  and  drivers/pilots  and 
gunners  for  the  following  vehicles:  AH-1,  AV-8B,  UH-1, 
CH-53,  LAV,  AAV,  and  M1A1.  Included  with  DVTE  for 
scenario  development  is  JSAF.  JSAF  is  the  main 
component  for  generation  of  interactive  scenarios  that 
support  multiple  simulated  entities.  DVTE  provides  an 
existing  distributed  simulation  environment  and  is  used 
within  different  configurations  of  the  MIX  Testbed.  The 
UVSIM  and  OCU  components  augment  DVTE  by 


providing  unmanned  systems  capabilities  for  training 
mixed-initiative  teams. 

3.2  Unmanned  Vehicle  Simulator  (UVSIM) 

The  UVSIM  application  is  used  to  generate  ground  or  air 
unmanned  systems  that  support  JAUS  as  a  command  and 
control  interface.  By  providing  a  standard  interface,  users 
only  need  to  learn  one  command  set,  reducing  training 
time.  JAUS  also  describes  a  set  of  services  that 
unmanned  systems  can  support.  For  example,  the  Global 
Pose  Sensor  service  provides  position  and  attitude 
information  about  a  vehicle.  Messages  within  the 
standard  allow  interfaces  to  query  the  system  for 
supported  services  letting  operators  discover  the 
capabilities  of  unmanned  systems  dynamically. 


Figure  2  -  Aided  target  recognition  scan 


With  the  two  automations  provided,  four  control 
conditions  can  be  produced  with  the  simulated  unmanned 
system:  teleoperation  with  no  AiTR,  teleoperation  with 
AiTR,  autonomous  navigation  with  no  AiTR,  and 
autonomous  navigation  with  AiTR. 


UVSIM  is  implemented  in  C++  using  Open  Scene  Graph 
and  the  JAUS++  library  created  by  the  ACTIVE 
Laboratory.  JAUS++  is  a  C++  implementation  of  JAUS 
that  provides  interfaces  for  common  services  in  addition 
to  a  complete  message  set.  Using  Open  Scene  Graph,  a 
3D  environment  is  generated  for  interactions  within  the 
environment  and  video  streaming.  UVSIM  can  be  run  in 
a  distributed  manner  with  single  or  multiple  unmanned 
systems  on  each  networked  machine  depending  on 
hardware  capabilities. 

UVSIM  can  operate  with  existing  HLA,  simulations  like 
DVTE,  using  a  JAUS/HLA  bridge,  and  also  supports 
standalone  capabilities.  Through  an  Extensible  Markup 
Language  (XML)  configuration  file,  users  can  add  static 
models  within  the  simulated  environment.  Any  3D  model 
supported  by  Open  Scene  Graph  can  be  loaded  and  placed 
at  a  pre -determined  location  for  creation  of  controlled 
conditions  between  missions. 

Each  simulated  vehicle  supports  a  common  set  of  services 
for  providing  pose  and  video  information,  but  do  not 
support  the  same  level  of  automation.  The  UGS 
simulation  in  the  current  implementation  of  MIX  supports 
two  types  of  automation:  waypoint  navigation  and  aided 
target  recognition  (AiTR).  Waypoint  navigation  allows 
operators  to  send  a  pre-planned  mission  route  to  the 
vehicle.  The  UGS  will  follow  the  given  route,  pausing  at 
designated  checkpoints  in  the  mission  plan.  The  AiTR 
pans  the  vehicle’s  camera  (up  to  360  degrees),  scanning 
for  targets  within  the  environment.  The  resulting  scan 
identifies  a  list  of  all  targets  (friendly  and  unfriendly) 
within  the  scene  as  shown  in  Figure  2. 


3.3  Operator  Control  Unit  (OCU) 

The  Operator  Control  Unit  is  the  primary  interface  used 
for  interactions  with  unmanned  systems  used  for 
reconnaissance,  surveillance,  and  target  identification,  and 
replicates  the  look  of  the  Tactical  Control  Unit  developed 
by  General  Dynamics  and  ARL  (General  Dynamics, 
2008).  It  is  also  developed  in  C++  with  Open  Scene 
Graph  and  the  JAUS++  library  for  communication.  It 
contains  a  “gods-eye”  view  of  the  map  showing  the 
current  location  of  all  unmanned  systems  available  for 
control.  An  asset  summary  list  allows  users  to  select 
specific  unmanned  systems.  A  navigation  bar  allows  for 
zooming  in/out  and  movement  across  the  map.  There  are 
four  main  screens  that  can  be  selected  from  the  toolbar  at 
the  bottom  of  the  screen:  Teleoperation,  Reconnaissance, 
Surveillance,  and  Target  Acquisition  (RSTA),  Video,  and 
Map. 


Figure  3  -  Operator  control  unit 

When  an  asset  is  selected  and  the  Teleoperation  screen  is 
selected,  users  can  remote  control  the  vehicle  using  an 
attached  joystick.  Video  from  the  vehicle  is  displayed  on 
screen  in  addition  to  the  “gods-eye”  view  of  the  map.  The 
Video  screen  looks  identical  to  the  teleoperation  screen; 
however  users  are  not  in  direct  control  of  the  vehicle. 


From  this  screen,  users  can  monitor  the  environment 
while  a  vehicle  is  navigating  autonomously.  The  Map 
screen.  Figure  3,  is  used  to  plot  mission  plans  for 
unmanned  assets  capable  of  autonomous  navigation. 
Finally,  the  RSTA  screen,  Figure  4,  displays  all  AiTR 
scans  received  from  an  unmanned  asset.  From  this  screen 
operators  can  send  an  AiTR  command,  place  markers  on 
the  map  representing  identified  targets,  and  send  reports 
containing  threats  detected. 


Figure  4  -  RSTA  screen 

As  shown  in  Figure  4,  the  RSTA  screen  contains  a  list  of 
all  AiTR  scans  received  on  the  left  side.  Users  select  an 
AiTR  scan  from  this  list  which  results  in  the  larger  image 
being  shown  at  the  top,  along  with  all  identified  targets, 
called  chips,  shown  on  the  bottom.  The  center  of  the 
screen  displays  the  “gods-eye”  view  of  the  map.  When  a 
chip  is  selected,  a  red  box  is  shown  in  the  larger  scan 
image  to  show  where  the  target  is  in  the  scene.  Using  a 
menu  on  the  right,  operators  can  identify  chips  as 
friendly,  neutral,  or  enemy  targets,  and  place  an  icon  on 
the  map. 

Mission  plans  developed  using  the  OCU  designate  a 
series  of  waypoints  and  checkpoints  that  a  vehicle  must 
navigate.  Checkpoints  indicate  locations  where  the 
vehicle  should  pause  in  the  mission  plan  so  that  additional 
reconnaissance  and  surveillance  tasks  can  be  performed. 
If  desired,  a  checkpoint  can  also  be  designated  as  an 
AiTR  location.  Whenever  an  unmanned  system  that 
supports  AiTR  reaches  one  of  these  checkpoints  it 
automatically  performs  the  scan  and  sends  the  results 
back  to  the  OCU.  A  vehicle  will  remain  at  a  checkpoint 
until  given  a  resume  command. 

To  simulate  secondary  and  tertiary  tasks  an  unmanned 
system  operator  may  perform,  the  OCU  also  has  the 
ability  to  play  audio  cues  for  participants  to  respond  to. 
These  audio  queues  are  pre-recorded  messages  that  can  be 


triggered  when  an  unmanned  system  enters  a  region  of  the 
map  or  a  timing  event  occurs.  The  triggering  criteria  and 
audio  files  to  play  can  be  configured  using  an  XML 
settings  file.  An  additional  visual  monitoring  task  can  be 
run  in  conjunction  with  the  OCU  for  experimentation 
purposes  (Figure  5).  During  scenario  execution,  the 
visual  monitoring  program  displays  a  series  of  gauges  that 
are  constantly  in  motion.  Based  on  timing  in  a  settings 
file,  an  event  is  generated  which  causes  one  or  more  of 
the  gauges  to  enter  an  upper  or  lower  limit.  By  pressing  a 
reset  button  the  gauges  will  return  to  normal  levels.  This 
task  can  also  be  used  concurrently  with  the  audio  queues. 
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Figure  5  -  Visual  monitoring  task 

In  addition  to  the  capabilities  for  RSTA  using  the  OCU, 
data  logging  capabilities  also  exist.  The  OCU  data  logger 
records  all  user  interactions  such  as  text  fields  entered, 
buttons  pressed,  scenario  events  (e.g.  unmanned  system 
position  changes),  and  also  performs  a  screen  capture. 
This  information  allows  researchers  to  keep  track  of  what 
tasks  a  user  was  most  heavily  involved  in  and  know  what 
the  environment  looked  like  from  the  user’s  perspective  at 
any  point  in  time. 

The  OCU  data  logger  is  also  capable  of  collecting  data 
from  neurophysiological  sensors.  This  information  is 
time-stamped  so  that  it  can  be  synchronized  with 
application  events.  For  example,  if  an  eye  tracker  is  used 
it  is  possible  to  identify  what  area  of  the  screen  a  user  was 
focused  on  when  an  audio  queue  or  other  event  is 
triggered.  An  application  interface  for  sending 
neurophysiological  data  to  the  OCU  for  logging  allows 
for  the  addition  of  new  sensors  at  a  later  date  without 
needing  to  modify  the  OCU  source  code.  Currently,  the 


OCU  incorporates  eye  tracking  and  heart  rate  monitoring 
data  for  measuring  a  user’s  workload. 

4.  MIX  SCENARIOS 

The  functionality  provided  by  the  MIX  Testbed  makes 
experimentation  in  HRI  and  mixed-initiative  teams 
possible.  Depending  on  the  type  of  research  or  training 
desired,  the  test  environment  can  be  reconfigured  in 
multiple  ways  due  to  its’  distributed  nature. 

4.1  Mixed-Initiative  Team  Training 

Using  all  components  of  the  MIX  Testbed  it  is  possible  to 
perform  scenarios  using  mixed-initiative  teams.  With  the 
use  of  DVTE,  UVSIM,  and  the  OCU  a  scenario  can  be 
assembled  where  an  unmanned  vehicle  operator  can 
provide  reconnaissance  information  to  a  FiST  team  during 
a  call  for  fire  exercise.  A  simulated  unmanned  air  system 
(UAS)  or  UGS  can  be  generated  by  UVSIM  in  order  to 
provide  video  information  from  the  virtual  battlefield. 
With  this  configuration,  team  training  can  be  conducted  to 
teach  the  correct  procedures  for  interactions  between  the 
operator  of  an  unmanned  asset  and  other  team  members. 

4.2  Human  Robot  Interactions 

Current  research  at  the  ACTIVE  Laboratory  using  the 
MIX  Testbed  involves  the  affects  of  automation  on 
situational  awareness  and  user  workload.  A  standalone 
configuration  of  the  OCU  and  UVSIM  applications  was 
used  where  participants  were  required  to  perform  a 
reconnaissance,  surveillance  and  target  identification  task 
using  a  simulated  UGS  and  the  OCU.  Participants  would 
monitor  the  environment  using  the  video  stream  from  the 
UGS,  detecting  unfriendly  targets.  In  all  conditions 
participants  performed  four  identical  missions,  however 
the  levels  of  automation  (e.g.  waypoint  navigation 
without  AiTR,  teleoperation  with  AiTR)  were  varied  in 
each  mission.  Additional  tasks  such  as  audio  cues  and 
visual  monitoring  of  gauges  were  used  to  vary  user 
workload.  Using  the  data  logging  capabilities  of  the 
OCU,  performance  metrics,  heart  rate,  and  eye  gaze  were 
recorded  to  identify  user  workload,  performance,  and 
situational  awareness  under  the  different  conditions. 

4.3  Unmanned  Systems  Operations 

With  the  incorporation  of  JAUS  as  a  primary  command 
and  control  interface,  the  MIX  Testbed  provides  a  means 
for  training  users  how  to  operate  unmanned  systems. 
With  JAUS,  the  OCU  is  also  capable  of  manipulating  real 
unmanned  systems,  making  it  an  ideal  environment  for 
learning  the  proper  mission  procedures  within  a 
simulation  before  moving  to  the  real  thing.  Moreover, 
JAUS  is  independent  of  hardware  and  technology,  letting 
users  learn  a  single  interface  for  controlling  air,  ground. 


surface,  or  undersea  unmanned  systems  reducing  training 
time  and  costs.  Users  would  still  be  required  to  learn  the 
differences  between  platforms;  however  this  would  be 
required  regardless  of  the  interface. 

CONCLUSION 

In  this  paper,  the  design  and  implementation  of  a 
simulation  environment  for  research  and  training  with 
unmanned  systems  is  described.  The  system  design 
includes  the  ability  to  incorporate  existing  distributed 
simulations  that  are  augmented  by  the  addition  of  an 
unmanned  vehicle  simulation  and  operator  control  unit. 
The  MIX  Testbed  is  an  implementation  of  this  design 
which  incorporates  DVTE  for  FiST  and  CAS  training  and 
is  enhanced  by  the  addition  of  unmanned  vehicle 
simulator  and  operator  control  unit. 

The  UVSIM  application  is  capable  of  generating  both 
unmanned  ground  and  air  systems  that  support  JAUS  as  a 
primary  command  and  control  interface.  Different  levels 
of  automation  are  supported  by  the  simulated  unmanned 
ground  vehicle  which  include  waypoint  navigation  and 
aided  target  recognition.  An  OCU  is  used  to  interact  with 
simulated  unmanned  systems  generated  by  UVSIM.  In 
addition  to  controlling  unmanned  systems,  the  OCU 
incorporates  data  logging  for  user  generated  events  and 
neurophysiological  sensors.  Several  scenarios  involving 
the  MIX  Testbed  are  presented  which  demonstrate  how 
different  configurations  can  be  used  for  training  and 
experimentation  in  HRI  and  mixed-initiative  teams. 

It  is  our  hypothesis  that  the  incorporation  of  the 
neurophysiological  sensors  and  performance  metrics  into 
the  testbed  can  lead  to  closed-loop,  adaptive  automation 
in  the  context  of  HRI.  As  a  user  interacts  with  the  MIX 
Testbed,  the  system  would  respond  to  improve  user 
performance  or  mitigate  poor  performance.  In  addition, 
the  system  responses  can  be  customized  to  an  individual, 
thereby  “learning”  strengths  and  weaknesses.  Future 
efforts  using  the  MIX  Testbed  will  incorporate  algorithms 
for  real  time  analysis  of  the  user’s  state  which  will 
facilitate  a  closed-loop  adaptive  system. 
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